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ABSTRACT 


Background: Registered Reports are publications in which 
study proposals are peer reviewed and pre- 
accepted before the study is run. Their adoption 
in other disciplines has been found to promote 


research quality and save time and resources. 


We offer a brief introduction to Registered Re- 
ports and their expected benefits for visualization 
research. We then report on a survey on the at- 
titudes held in the Visualization community to- 
wards Registered Reports. 


Objectives: 


Method: We ran an online survey that was open from Octo- 
ber 7, 2021 until October 24, 2021. We received 


105 responses from visualization researchers. 


Results: Most respondents expressed interest in Registered 
Reports as an additional submission format. Ju- 
nior researchers as well as researchers with prior 
experience of preregistration tended to be more in- 
clined to adopt this format. However, qualitative 
feedback highlights several perceived drawbacks 
and worries about Registered Reports, such as 
low suitability for some types of research, and 
increased workload for reviewers and authors. 


While many respondents expressed interest in 
(and sometimes enthusiasm for) Registered Re- 
ports, many also voiced concerns that should be 
considered and discussed in the community. 


Conclusion: 


Reproducibility: All materials are available at osf.io/4nrma/. 


Index Terms: Human-centered computing—Visualization—Visu- 
alization techniques—Treemaps; Human-centered computing— 
Visualization— Visualization design and evaluation methods 


1 INTRODUCTION 


Publishing scientific papers is the primary means of communicating 
results, ideas, and tools to the public and scientific community. 
The publication of new scientific communications in journals or 
conferences is subject to peer-review which eventually decides if 
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the communication is worth publishing. However, traditional peer- 
review has been found to be sub-optimal [10, 48,50]: it is hardly 
objective and there are many biases hindering the process, including 
a bias for positive results [15]. To address some of these issues and 
after facing the replication crisis [35], many fields tried improving 
their research and publishing methodology. Out of these efforts, 
Registered Reports emerged. According to the Center for Open 
Science, a Registered Report is: 


“a form of empirical journal article in which meth- 
ods and proposed analyses are preregistered and peer- 
reviewed prior to research being conducted. High quality 
protocols are then provisionally accepted for publication 
before data collection commences. This format of article 
is designed to reward best practice in adhering to the 
hypothetico-deductive model of the scientific method. It 
eliminates a variety of questionable research practices, 
including low statistical power, selective reporting of 
results, and publication bias, while allowing complete 
flexibility to conduct exploratory (unregistered) analyses 
and report serendipitous findings.” 


The benefits of Registered Reports have been discussed exten- 
sively in the literature (see Sect. 2), and this publication format has 
been advocated by empirical computer science [15]. However, to our 
knowledge, no visualization journal or conference offers Registered 
Reports as a publication format. Here, we (i) explain Registered 
Reports and their benefits to the visualization community, and (ii) 
report the results of a survey. This survey aims at better understand- 
ing the visualization community’s general awareness of Registered 
Reports, in how far such a publication model fits into community 
members’ habits and expectations, and in crowdsourcing potential 
issues which may need to be taken into account to adapt Registered 
Reports to the needs of the visualization community. 

We refer to our own article as a quasi Registered Report, because 
it tries to be as close as possible to a Registered Report but it has 
not been submitted to a venue with a proper two-stage Reviewing 
Process already in place. Even then, to the best of our knowledge, 
this article is the closest to a Registered Report published in a vi- 
sualization venue so far. The present document is the second-stage 
version, with all survey results and discussions. The first-stage ver- 
sion, which has been accepted and presented at the alt. VIS venue 
before we ran the survey, is available as a preprint online [11]. 


2 REGISTERED REPORTS 


The concept of Registered Report has been conceived more than a 
decade ago. Before we summarize here the hypothesized benefits 
and the evidence found to support them, we first quickly explain 
the differences between Registered Reports and preregistrations, 
because the terms are quite similar and can be easily confused. 
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2.1 Registered Reports vs. Preregistrations 


Preregistering a study consists of uploading a complete data col- 
lection and analysis plan to an online repository that time-stamps 
it and conserves it [37,39]. This plan (called a preregistration or 
sometimes a registration) is submitted before the study is conducted. 
Its goal is to make research more transparent and more trustwor- 
thy by drawing a clear line between confirmatory and exploratory 
analyses, and by preventing unintentional inflation of false-positive 
findings through analytical flexibility [9, 21,38]. After the study is 
conducted, the researcher includes a link to the online preregistration 
in their paper, as a proof that they have followed a pre-specified plan. 
Deviations from the plan are allowed as long as they are explained 
in the paper. Preregistered studies can, by default, be submitted as 
regular papers and thus get peer-review feedback only after the study 
is already conducted. 

A Registered Report is similar to a preregistration in that re- 
searchers must commit to a complete data collection and analysis 
plan before they run their study. However, a Registered Report is 
not just an analysis plan: it takes the form of a paper with partial 
content, called Stage 1 Registered Report, that is submitted for peer 
review instead of being uploaded to an online registry. This Stage 1 
submission usually contains an introduction, related work as well 
as the methodology, including analysis plan. Reviewers can make 
comments about the study plan and ask for changes before the study 
is conducted. In addition, a study submitted as a Registered Report 
is conditionally pre-accepted for publication before the study is con- 
ducted, which is not possible with a preregistered study submitted 
as a regular paper. 

Because study preregistration does not require any change to 
standard reviewing procedures, it can be readily adopted and used 
in any venue. Authors in the Visualization community have indeed 
begun to publish preregistered studies (e.g., [7,27, 28, 33,42, 52]). 
In contrast, Registered Reports are not yet supported. 


2.2 Benefits of Registered Reports 
2.2.1 Mitigating the File-Drawer Effect 


A Registered Report is inherently result-agnostic: decisions to ac- 
cept or reject publications are solely based on the relevance of the 
research questions and the proposed methodology to study these. As 
such, Registered Reports are expected to mitigate the bias towards 
the publication of positive findings, also known as the file-drawer 
effect. Not only does this bias hide interventions that do not work, 
it negatively impacts the trustworthiness of published results and 
can harm the credibility of a field [23,49]. Initial investigation in 
psychology [44] has found some evidence that Registered Reports 
can indeed mitigate publication bias: Registered Reports presented 
only 47% positive results while comparable classical submissions 
presented an outstanding 96% positive results. 


2.2.2 Avoiding Questionable Research Practices 


By design, both Registered Reports and pregistrations are meant 
to prevent questionable research practices. In particular, they ren- 
der HARKing (hypothesizing after the results are known) [16] im- 
possible, and help prevent selective reporting and p-hacking [15]. 
However, in addition to preregistrations, Registered Report formats 
involve peer review prior to data collection, which gives them an 
additional benefit: the experimental design and methods can be 
improved upon through the recommendations made by reviewers. 


2.2.3 Improving Quality and Reducing Scientific Waste 


The reviewing of Registered Reports focuses on the authors’ research 
questions and their importance, as well as the proposed methodol- 
ogy. The iterations between authors and reviewers before the study 
is conducted are likely to improve experimental design [6], help 
correct a wrongfully chosen research outcome/question, and detect 
errors or imprecision in data analysis. These iterations all take place 


before any data collection begins and thus before any scientific re- 
source is used, be it participants that are enrolled, money spent, 
or devices/doses used. For this reason, Registered Reports have 
been said to help reduce waste of scientific resources [9, 13, 14]. 
In addition, they could help reduce the norm of submitting already 
completed but potentially flawed studies to many different journals 
until they get accepted [24, 36]. Even when only considering the 
time spent reformatting scholarly articles from one journal template 
to another [30], Registered Reports would already save precious 
scientific time. 


2.3 Perceived Quality and Novelty 


While Registered Reports do seem to increase research quality and 
mitigate many of the biases of the publishing system, some have 
voiced concerns that they could lead to scientists pursuing less 
novel or interesting research questions [8, 17]. However, initial 
evidence would seem to suggest that such fears are unfounded, 
and that Registered Reports are perceived by scholars as being 
statistically indistinguishable from regular papers in terms of novelty 
or creativity, while being at the same time perceived as more robust 
and of higher quality [46]. 


3 REGISTERED REPORTS FOR VISUALIZATION RESEARCH: 
WHY AND How 


Today, according to the Center for Open Science, more than 300 
scientific venues offer the option to submit Registered Reports.! 
However, to the best of our knowledge, no visualization or even 
human-computer interaction venue offers the possibility to submit 
or review Registered Reports. Cockburn et al. [15] argued that the 
adoption of Registered Reports at major human-computer interaction 
venues could be daunting, but that reviewing workflows could easily 
be adapted to allow it. 

Visualization papers come in many different types [31,34], and 
not all of them report a user study. Those that do not include a user 
study would not benefit from the addition of Registered Reports 
as a publication type since no experimental design is included in 
them. Such manuscripts could continue to be submitted as regular 
papers and reviewed accordingly. On the other hand, papers with 
any form of empirical evaluation could benefit from being reviewed 
as Registered Reports. This form of review could be optional, and it 
could be up to the authors to choose whether or not they want their 
work to be reviewed as a registered report. 

As an example, consider an author who wants to publish a novel 
visualization technique with an evaluation. They would submit a 
first-stage Registered Report that contains an introduction explaining 
the problem the technique is intended to solve, an overview of related 
work, a full description of the novel system or technique (ideally with 
an implementation available for reviewers to try), and a final section 
detailing how the empirical evaluation will be conducted, what data 
will be collected and how the data will be analyzed. Discussions 
and conclusion sections are left out. As another example, if an 
author wants to compare existing techniques or conduct a study that 
does not involve any new technique, their Registered Report would 
contain all the sections previously mentioned except the section on 
the novel system or technique. As reviewers may ask for changes 
in the experimental design, it would be futile to commence data 
collection before the reviews are sent to the authors. Reviewers 
might even suggest small improvements to the techniques tested. 

For reviewers to be able to evaluate the methodology proposed 
by authors, the empirical evaluation section needs to be described 
as precisely as possible, much like a study preregistration [39]. 
Authors should include: research questions and hypotheses, detailed 
experiment design with all independent and dependent variables, 
sample size, and data analysis plan. Adding drafts of plots and tables 


‘https ://www.cos.io/initiatives/registered-reports 
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with fictional data can be very helpful to reviewers. As an example, 
the first-stage version of the present quasi Registered Report can be 
found at osf.io/4nrma/ [9]. 


4 SURVEY OF VISUALIZATION RESEARCHERS 


We describe below our research questions and hypotheses as well as 
our survey questions, its administration and the data analysis. 


4.1 Research Questions and Hypotheses 


We wanted to understand visualization researchers’ knowledge of 
and attitudes towards Registered Reports. We set out to answer if 
the visualization community knows about Registered Reports, feels 
confident about submitting and reviewing them, and is ultimately 
interested in adopting them for submissions. We anticipated that 
interest and confidence might be dependent on the respondent’s level 
of adoption of open science practices (e.g., preregistrations [40]), 
and perhaps also on their years of expertise in the field. 


4.2 Administration and Ethics Approval 


The survey was presented as an online form, and participation was 
voluntary. Participants were informed about the purpose of the sur- 
vey and its approximate completion time (based on pilot studies) 
before they started answering. No personal information was col- 
lected. No compensation was offered but participants were given 
the link to our OSF repository so they could later check the results 
of the survey. 


4.3 Recruitment 


We shared our survey on social media (Twitter, Slack,...) and by 
email to past and present collaborators. We invited colleagues 
through word of mouth and informal sharing. Based on a previ- 
ous similar survey of the visualization community, we anticipated 
that our survey would gather at least 50 answers. The final number 
of participants was 105. 


4.4 Timing 


Once our first-stage quasi Registered Report had been reviewed by 
the alt. VIS workshop organizers, we iterated on our survey design 
and our data analysis plan. We then submitted our study protocol for 
ethics approval. We started data collection after we obtained ethics 
approval. On the day of the workshop, we presented our preliminary 
results. We stopped data collection the night of the alt. VIS workshop, 
at 23:59 AoE time (October, 24 2021). We then wrote this second- 
stage Registered Report, which includes an analysis and discussion 
of all our 105 responses. In order to remain as close as possible to a 
Registered Report, we have sent the last revision to the workshop 
organizers (see letter to workshop organizers submitted together 
with the stage 1 version of this article on our OSF repository). Two 
organizers provided a light second-stage review to approve of our 
final manuscript. 


4.5 Survey and Analysis Plan 


We describe below the content of our survey and our analysis plan. 


4.5.1 Survey 


The survey we used is available on our OSF repository 
(osf.io/4nrma/). It consists of five pages. The first page details 
the aim of the survey and asks for the participant’s informed consent. 
The second page aims to gather participants’ research experience 
through Likert items, asking them, for example, how long they have 
been active in the visualization community or their knowledge and 
use of preregistrations. The third page asks participants if they know 
what a Registered Report is, before giving, on the fourth page, the 
definition of Registered Reports according to the Center for Open 
Science. After having been explained what Registered Reports are, 
participants were invited to tell us if this definition corresponds to 
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Figure 1: Likert Plots for the four attitude questions. 


what they thought a Registered Report was, through Likert Items. 
We then invited them to use four Likert items to assess: 


Q1 how helpful they think it would be to have their study design 
reviewed before data collection, 


Q2 how willing they would be to submit a Registered Report, 
Q3 how willing they would be to review a Registered Report, 


Q4 how interested they would be in having Registered Reports as 
a publication option in the visualization community. 


All of the questions above are complemented with an optional text 
field where participants can explain their answers and make com- 
ments if they wish to. 


4.5.2 Data Analysis Plan 


The detailed analysis plan that we submitted as a first-stage Regis- 
tered Report [11] is available on our OSF repository (osf.io/4nrma/). 
It included a qualitative and quantitative analysis of simulated data. 
We also planned to report the results of the research experience ques- 
tions on page 2 as descriptive statistics. Any analysis not included in 
the first-stage Registered Report is reported below as complementary 
analysis. 


4.6 Changes to the Plan 


During late pilot testing, we realized that some participants might 
wrongly report knowing what a registered report is, for example if 
they read it as a synonym of a preregistered study. Therefore, after 
explaining what the term means, we added a question asking how 
well the explanation captures what they thought a registered report 
was. Response analyses are reported in Sect. 5.2. 

Concerning participant recruitment, our Stage 1 submission states 
that we planned to share our survey on social media (Twitter, 
Slack,...) and by email to past and present collaborators. We thus 
initially only invited colleagues through word of mouth and informal 
sharing. However, we realized that this approach would likely only 
reach a small subset of all potential participants considered eligible 
according to the criteria indicated in the survey. We therefore de- 
cided to additionally extracted 796 email addresses of from articles 
published at VIS and sent an email invitation to all past authors. 

All other changes were trivial corrections to the wording of the 
survey and to the analysis code. 


5 RESULTS 


We received a total of 105 responses to our survey, which we analyze 
here. Our sample is comprised of 56 people with “10 years or more” 
of experience (53%), 28 with “5 to 10 years” (27%), and 21 with 
“less than 5 years” (20%). 
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Figure 2: Mean responses to the four attitude questions depending 
on years of experience in visualization research. Error bars: 95% Cls. 
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Figure 3: Mean responses to the four attitude questions depending 
on the number of preregistrations submitted. Error bars: 95% Cls. 


5.1 Planned Quantitative Analysis 


We report a Likert plot showing responses to each of the four at- 
titude questions in Fig. 1. With respect to our first question (Q1) 
on the usefulness of early reviews, only 5 respondents (5%) found 
early reviews as provided by Registered Reports “not helpful at all”. 
Responses to Q2 and Q3 show that most respondents (75 respon- 
dents, 71%) are rather interested and willing to submit Registered 
Reports and that many of them (36 respondents, 34%) are willing to 
review them. The slightly less positive answers for Q2 (reviewing), 
might indicate that finding reviewers for Registered Reports could 
be, at least initially, slightly difficult if the visualization community 
adopted Registered Reports. That being said, the mismatch does 
not seem to be too large as a significant number of participants (50 
respondents, 48%) seem to be neutral about reviewing Registered 
Reports, and the perception of this new submission format may 
change once they become more common. Finally, responses to the 
fourth question (Q4) show that a majority of our respondents are 
interested in having Registered Reports as an alternative publica- 
tion model (66 participants, 63%). Note that although responses 
to Q1 appear more negative than responses to Q2, Q3, and Q4, the 
response options were different, with a more positive framing for 
Q1 (e.g., compare “Somewhat helpful” for Q1 with “Reluctant” for 
Q2). 

To check whether attitude depends on seniority (operationalized 
as years of experiences in visualization research), or on current 
adoption of open science practices (operationalized as the num- 
ber of preregistered studies [40] previously submitted), we report 
average responses to the four attitude questions as a function of 
seniority (Fig. 2) and number of preregistrations submitted (Fig. 3). 
In both figures, we report the mean response with 95% bootstrap 
confidence intervals. While the use of means to summarize ordi- 
nal responses is not universally accepted [43], it is a common and 


methodologically-defensible practice [32,43]. We interpret our re- 
sults with an estimation approach [20]. 

In Fig. 2, we see that younger researchers are overall slightly 
more positive in their answers to the four attitude questions than 
more experienced researchers. The difference is very clear between 
junior researchers (< 5 years of experience) and the most senior 
ones (> 10 years), but evidence that there is a difference on average 
is weaker for adjacent categories. In Fig. 3 we have good evidence 
that people with no prior preregistration experience are slightly more 
negative in their answers than people with some experience (1 to 5 
preregistrations submitted). For people with 6+ preregistrations (6 
respondents out of 93), intervals are too wide to conclude due to the 
small sample size. 


5.2 Additional Quantitative Analyses 


Here we report responses to the non-attitude questions like knowl- 
edge and use of preregistration and registered reports (Fig. 4). First, 
concerning familiarity with preregistrations, the answers we ob- 
tained suggest that overall, knowledge of preregistration varies a lot 
among our respondents. Indeed, 39 participants (37%) responded 
“T have a good understanding of what it is” , while 38 (36%) ticked 
“I have a vague notion of what it means”, and 28 (27%) reported 
never having heard of it. Lack of familiarity could be explained 
by the breadth of visualization-related publications for which pre- 
registrations are not useful. Concerning familiarity with Registered 
Reports, the majority of our sample reported having little to no 
knowledge. Indeed, 46 participants (44%) reported having “never 
heard of it”, and 41 (39%) said they had “a vague notion of what it 
means.” Only 14 (13%) reported having “a good understanding of 
what it is”. Four participants (4%) answered that they had “written 
and submitted [one] before.” Since no visualization venues offer 
Registered Reports, these answers could be explained either by the 
fact that they submitted one in another field of research, or that 
they mistook Registered Reports for preregistrations. Related to this 
point, we asked participants if the definition of a Registered Report 
corresponded to what they expected. The answers to this question 
were: 13 (12%) “Not at all”, 12 (11%) “Not very well”, 47 (45%) 
“Reasonably well”, 24 (23%) “Very well”, 9 (9%) “Perfectly”. These 
answers are consistent with our hypothesis that some people mistook 
a preregistration for a Registered Report. 


5.3 Qualitative Analysis 


Our survey gave respondents the opportunity to provide additional 
explanations about the four attitude questions in the survey. We 
conducted a thematic analysis of these answers. Two coders of 
the author team analyzed responses independently to categorize 
them based on their themes. A third coder was involved to resolve 
non-matching categorization of answers. 

Our complete list of codes is reported in Table 1, with frequencies 
of appearance. If a code appears multiple times in a respondent’s 
answers, it is counted only once. The maximum count is 55, which 
is the number of respondents (out of the total 105) who provided 
qualitative data. The other respondents filled out the form but did 
not enter any explanatory text. 

The comments we received on questions Q1-Q4 predominantly 
raised concerns regarding Registered Reports but also highlighted 
some of their benefits. Here we highlight the major themes we found 
and comment on the most important in the discussion section. 


5.3.1 Comments on the Benefits of Registered Reports 


As a major benefit, respondents highlighted the possibility of Regis- 
tered Reports to improve research quality. Respondents mentioned 
that they would welcome early help and feedback (9x Q1, 4x Q2) 
and expect to see an increase in the quality of their own research (5x 


Q1, 4x Q2). 
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Table 1: Full list of codes from the qualitative analysis regarding our questions about registered reports (RR). 


Code # Responses (out of 55) 
Comments on Q1: 46 Seem S 
How helpful would it would be to have your study design reviewed before data collection? 

Reasons given why RRs would be useful 19 munsassaueuausausen 


One can receive help and insight from others (early) 

RRs will increase research quality 

RRs will reduce risk and wasted resources 

RRs fit my work or a certain type of visualization work/process 

RRs would help junior or researchers new to a field 

Early review can increased confidence in own research 

Early feedback & planning are already part of our research process 

RRs allow to publish negative or inconclusive results 

Early feedback reduces risk of unexpected reviewer feedback 
Reasons given why early RRs would not be helpful 2 

RRs are not suitable for certain types of vis / hci work 

RRs increase workload for authors and risk delaying publication 

RRs make adjustments to research more difficult 

RRs increase workload for reviewers and organizational overhead 

Current practices can already ensure sufficient research quality 

There is no required expertise & guidelines yet in the community 

RRs create the risk of reviewers stealing ideas 

Potential for poor reviews remains with RRs 

For reviewers it is hard to identify problems without seeing the final study 
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RRs may give a false sense of security. Issues may appear after early review nu 
RR reviewers might steer research in a bad direction = 
Comments that were not clearly for or against finding RR helpful lo 
Researchers get feedback informally now lo 
Comments regarding Q3: How willing would you be to review a RR? 36 SSSR EERE 


Reason for being willing to review a RR 9 m 
I would like to help improve research quality 
It is fulfilling to give advice early 
Early review might be easier 
Early review allows to focus reviews on other aspects 
Reason against being willing to review a RR 25 m 
It takes more time, increases workload 1 
I do not have the expertise to assess study designs/statistical analyses 
An early reviewer has a higher responsibility for others’ research projects 
I don’t want to be involved with RR 


we 
= 
= 
= 
= 


= = = NNN FUER 
O 
CI 


LL] 
I cannot see the benefit of RRs == 
I would not want to dictate my opinions on methodology to others = 
RRs might incentivise sloppier second round reviewing = 
Reviewing RRs is more difficult = 
Comments on Q2: How willing are you to submit a RR 31 muannssansonsensuansansennunnunnx 
Reason for being willing to submit a RR 8 = ma 


To improve the quality of my research 4 mamn 
To receive early feedback 4 mumn 
To make it easier to publish the research I do l a 
To open discussion of unpublished work lao 
To save resources l m 
Reason for not being willing to submit a RR 13 munsausausuus 


RR increase my workload 
Review quality for RRs might be too low 
RRs increase the time to get published 
I would need trust reviewer confidentiality 
Not clear if reviewer comments will be helpful 
RRs are not a fit for the work I do 
RRs do not seem useful for validating visualization designs 
Already have different ways to accomplish the goals of RRs 
I would submit if... 
.. the research I do matches 
. review quality is good 
.. the timing works out 
.. it doesn’t increase the amount of work 
.. I will not receive a random stage 2 rejection 
. -reviews are helpful 
.. there will be credit for reviewers 


4 
4 
4 
2 == 
2 
2 
1 
1 


Comments on Q4: 
How interested are you in having RRs as a publication option in the visualization community? 


Reasons for interest in RRs as submission category 7 mannana 
RRs would benefit the community 3 om 
RRs would increase scientific rigor 2 ma 
RRs allow publishing negative results lo 
RRs are the right thing to do lao 
If RRs are introduced as a complementary publication model l m 

Reason against interest in RRs as submission category 10 munsassaus 
RRs can have a negative impact on some parts of Vis research 3 mma 
It is unclear if RRs are practically realizable 3 om 
RRs increase the community’s workload 2 mu 
RRs raise confidentiality concerns regarding ”’stealing” of ideas 2 ma 
It is unclear if there are enough qualified reviewers l m 
There is a risk of breaking the current reviewing system la 


This manuscript was presented at alt.VIS, a workshop co-located with IEEE VIS 2021 (held virtually) 


Preregistrations 


| have a good 
understanding 


eer whetit neon: MS 
of what it means 


0 10 20 30 40 50 
Registered Reports 


I have written and 
submitted one 


Ihave a good 
understanding of 


wnat ees SES 
of what it means 


I never heard of it 


Perfectly a 
Not very well a 
Not at all | 
0 10 20 30 40 50 


Figure 4: Responses to the definition questions. 


“I'm supportive of the idea of registered reports: they 
provide a sanity check on the soundness of a proposed 
study and can lower barriers to engaging in experimental 
research to people just getting involved in such study.” 


Four respondents (Q2) mentioned that they would be happy to help 
improve research quality as a reviewer of registered reports and three 
that they would find it fulfilling to give advice early (Q2). 


“I rather review a registered review and give authors 
feedback early on than telling authors that their study 
is not ”useful” afterwards and feeling bad because they 
probably have spend time, money, and resources in con- 
ducting a study which is leading nowhere.” 


Five responses similarly mentioned the potential for registered re- 
ports to save resources (time, money, ...) in the research process 


(4x Q1, 1x Q2). 
5.3.2 Concerns Raised Regarding Registered Reports 


Despite some respondents mentioning that they could save time and 
resources with Registered Reports, others saw time and workload as 
a risk. Specifically, responses mentioned an increased workload for 
authors and risk of delaying publication (8x Q1, 8x total Q2). 


“There are questions where the experimental methods 
are tricky (or even part of the contribution), in which case 
having additional input on the methods would be helpful. 
Other experiments are pretty straightforward, and having 
an expectation of prior methodological review would 
simply be yet another delay in the process of getting the 
results to the research community.” 


Others voiced concerns over increased workload for reviewers (6x 


Q1, 15x Q3). 


“My reviewing load is already high and I do not see how 
to reduce it. Thus, when I start a new line of reviewing 
activity, the question arises: How to get rid of some 
existing (reviewing) tasks.” 


Comments regarding added workload were sometimes coupled 
with concerns regarding reviewer expertise. Reviewer qualification 
concerns were evident in several responses (11x total Q1, 6x to- 
tal Q2, 4x Q3, 3x total Q4). Authors feared that their work may 
be steered in the wrong direction and that the community has no 
guidelines and procedures in place to deal with Registered Reports. 


“While most reviewers are reasonable, there are often re- 
viewers who believe the purpose of a review is to oppose 
one’s own view onto the authors.” 


From the reviewer’s perspective, several comments mentioned that 
it might be harder to review incomplete studies: 


“The direct acceptance condition worries me a bit as it 
is often hard to identify a problematic study just based 
on its description. Small variations in experimental in- 
structions or in an experimental apparatus can make a 
huge difference. As a reviewer, I may only realize a huge 
problem by looking on the reported results (or the raw 
data).” 


Several respondents (2x Q1, 2x Q2, 2x Q4) voiced fears over re- 
viewers being able to steal ideas for research to be conducted (i.e., 
scooping) and asked questions about the early publication of Regis- 
tered Reports to make them citable. 


“my question is regarding citing such a registered report 
and making sure others do not steal” the idea - does a 
registered report have a DOI that can be cited later on, 
this is a bit unclear.” 


The most common worry regarding Registered Reports, however, 
concerned the applicability to only certain types of VIS work (12x 
Ql, 2x Q2, 3x Q4). In particular the applicability to qualitative 
research was mentioned by several respondents: 


“Tt is the very nature of qualitative research that the 
subject of interest can adapt and change as the study 
progresses. Therefore the whole notion of preregistration 
is out of alignment with this approach to research.” 


“Furthermore, I am deeply concerned that efforts that 
make preregistration (and registered reports) an expecta- 
tion in vis will further marginalize qualitative research 
studies, which are already marginalized.” 


Similar to these concerns, other responses expressed a worry that by 
writing a Registered Report, the research would be constrained by 
external reviewers, or that no subsequent changes could be made to 
adjust and improve studies. 


“This could stifle approaches and innovation by discour- 
aging efforts early in the process.” 


“A preregistration is almost guaranteed to overtly con- 
strain the research to reinforce preconceptions.” 


A few respondents found registered reports simply unnecessary 
because they already used other mechanisms for collecting early 
feedback outside of a formal review process, for example through 
internal reviews (3x Q1, 1x Q2). 

Several respondents made conditional statements regarding their 
willingness to adopt and review Registered Reports that echo the 
concerns listed above. Respondents mentioned, for example, that 
they would submit Registered Reports if their research matches, if 
they can expect a high review quality, or if Registered Reports would 
mean not losing time or wasting effort in the research process. 
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6 DISCUSSIONS 


Our quantitative results highlight both interests but also worries 
about having Registered Reports available as a publication option 
for the visualization community. Some of the mentioned concerns 
are legitimate, while others may partly stem from a misconception 
of what a registered report ultimately is and means to achieve. We 
already discussed some concerns in Sect. 2 (e.g., that Registered 
Reports would be too time-consuming) and here we cover additional 
concerns that we consider important to address. 


6.1 Constraining Research and Creativity 


Some of the answers we obtained categorized Registered Reports 
as being constraining regarding innovation but also the research 
process itself. This parallels a perception from many researchers 
that study preregistration constrains research [18,37]. However, the 
idea of pre-committing to a data collection and analysis plan (both 
in Registered Reports and in preregistrations) is not to forbid any 
deviation from the initial plan, in particular if such a deviation would 
allow the protocol to eventually be more robust or the research to be 
more worthwhile, but rather to make sure that the deviations from 
the initial plan are transparently reported [37]. In fact, while the 
authors of this paper have preregistered their analysis plans in some 
of their visualization papers, they also have deviated from such plans 
on several occasions, but transparently reported deviations (see e.g., 
[45,51]). However, contrary to preregistration, Registered Reports 
get a conditional acceptance based on their initial plan, so while 
some deviation might be possible, the degree of deviation allowed 
may well be smaller. We discuss this issue further in Sect. 6.5. 

Some respondents worried that Registered Reports could hinder 
the novelty or creativity of produced research. Although this is 
possible, Soderberg et al. [46] found that reviewers in psychology 
and neuroscience judged Registered Reports as no less novel or 
creative than traditional papers, while rating Registered Reports as 
more rigorous and of higher quality overall. As such, there is some 
evidence that creativity in study design might not be impacted as 
much as feared by some respondents. However, creative visualiza- 
tion design work (as opposed to experiment design work) would 
likely not benefit from being submitted as a Registered Report due 
to the evolving nature of creative work. 


6.2 Risks of Stealth and Scooping 


Echoing past findings around perception of preregistration [47], 
some of our respondents highlighted their fears of a breach of con- 
fidentiality: they worried that reviewers might steal their ideas and 
potentially run the study before them. Such behavior in any review 
process would be considered academic misconduct (see for example 
the following retraction statement [3]) and can happen also with a 
traditional reviewing process. Once uncovered, this behavior will 
usually lead to retractions [4]. However, a simple way to avoid 
this issue is to preprint first-stage Registered Reports when they are 
submitted. This is also what we have done with this quasi Registered 
Report [11]. Preprints are time-stamped and allow researchers to 
claim intellectual precedence [19,22,41]. 


6.3 Unreasonable Reviewer Requests 


Some of our respondents mentioned that they feared unreasonable 
demands from their peers during the first stage of the reviewing 
process. There is indeed a risk that reviewers require changes to an 
experimental protocol that the authors consider unreasonable or mis- 
guided. In classical reviewing workflows, reviewers can influence 
or require changes to a statistical analysis, but not to an experiment 
protocol, since the study has already been conducted. If they do not 
like the way the study was done, reviewers can only ask the authors 
to highlight its limitations, or recommend rejection. However, the 
situation is not so different with Registered Reports, since authors 
are not obligated to make any changes they disagree with, and, if 


their manuscript is pending major/minor revision, they are free to 
explain their disagreements to reviewers through the summary of 
revisions and responses to reviewers. Should they fail to convince 
them, their paper will be rejected just as they would have likely been 
rejected had the study been already conducted. The authors could 
then try to submit their Register Report to another venue (should 
this format become more broadly accepted in visualization) or con- 
duct their study anyway and submit it as a regular paper. In such 
a case, the authors would have, in the worst case, just delayed the 
publication of their work, and will most likely have received at least 
some valuable feedback on their study design. Finally, opening up 
the reviewing process, by making the anonymous reviews public 
(and anonymous), might help mitigate the issue of unreasonable re- 
views [10], as it may increase scrutiny on both the Registered Report 
and its reviews, and therefore increase the chances that unreasonable 
demands are publicly flagged. 


6.4 Registered Reports and Qualitative Research 


In the answers to our survey, several researchers highlighted the 
potential incompatibility of Registered Reports with qualitative re- 
search. This is a legitimate worry considering that Registered Re- 
ports were initially designed, much like preregistration, for quantita- 
tive methods [12]. Given the relative novelty of Registered Reports, 
there is little literature on their possible benefits for qualitative re- 
search. However, there is some literature on the benefits and chal- 
lenges of preregistration for qualitative research [25, 26, 29]. Similar 
to Registered Reports for quantitative research, Registered Reports 
of qualitative work could also provide early feedback on the study 
design, prevent publication bias, and by forcing researchers to de- 
scribe their processes clearly they could promote transparency and 
increase the perception of rigor for qualitative work. 

However, the depth to which the data analysis procedures can be 
described for qualitative work will necessarily be different compared 
to quantitative research. As the researcher is the one producing sum- 
mary data and patterns—and not a specific statistical instrument— 
the processes of data analysis are not mechanical and the success 
of a data analysis hinges on the ability of the researcher and their 
invested time and effort to dig through emerging patterns. As such it 
is perhaps not a given (and this needs to be discussed further) that a 
Registered Report of qualitative work accepted in Stage 1 should be 
auto-accepted in Stage 2. 

It thus seems that more discussions are needed on the topic of 
the Registered Report format and how it could work for qualitative 
research. Noteworthy, some venues already use two-step submission 
processes for contributions which are not quantitative research, for 
example, the State of the Art Report (STAR) track at Eurovis. While 
at those there is no conditional acceptance after the first step, the ben- 
efits for research mentioned above in the qualitative responses likely 
apply there as well, for instance, benefiting from early feedback and 
reducing the risk of wasted resources. 


6.5 How much Deviation is Acceptable 


Although it is considered acceptable to deviate from a preregistration 
[37], a Stage 1 registered report that is approved by reviewers is 
essentially a conditionally accepted paper. Because the reviewers 
only approved the version of the protocol that they have seen, it is 
unclear what kind of changes would be deemed acceptable for a 
Stage 2 submission. For instance, we do not know if the changes 
we have made to our own survey (Sect. 4.6) would have disqualified 
our paper in an actual Registered Report reviewing process. We 
scanned through some of the journals that implement Registered 
Reports, and found that they rarely provide clear guidelines on what 
changes are acceptable between a Stage | and Stage 2 manuscript 
[1,2,5]. It seems that journals usually advise authors who want 
to make changes to contact the editor and to potentially resubmit 
for a Stage 1 reviewing before proceeding to data collection. If 
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Registered Reports were to be adopted in visualization research, it 
would be helpful to have clear guidelines explaining what changes 
can be considered minor and what changes require a re-submission 
as Stage 1. Regardless, this type of decision is inherently subjective 
and in many cases, the final decision will likely depend on who is 
reviewing. 


7 CONCLUSION 


We introduced and motivated the use of Registered Reports as a 
publication model for visualization studies, and reported on a survey 
attempting to better understand visualization researchers’ attitudes 
towards them. This article itself takes the form of a quasi Registered 
Report, which shares most of the characteristics of Registered Re- 
ports, except the alt. VIS reviewing workflow has not been designed 
with Registered Reports in mind. Despite the widespread adoption of 
this new publication model in a range of disciplines, and the tremen- 
dous benefits it can provide to all actors involved in certain types 
of research (authors, reviewers, journal editors), Registered Reports 
do not yet exist for visualization work. Our survey highlights that 
the community seems to be open to the idea of Registered Reports 
as an alternative publication model in visualization venues. The 
qualitative feedback we gathered also helped us understand some 
of the concerns around registration overall, some of which we have 
hopefully addressed in our discussions, and others that still need 
more in-depth work and discussion within our community. We hope 
this article will encourage journal editors, conference organizers, 
and authors in visualization to consider adopting Registered Reports 
as an additional publication format, and help them understand the 
opportunities and challenges in implementing them. 
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